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intended for use by 20 community colleges in California undergoing 
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Following introductory material describing the paper's background and 
the FIPSE project, the purposes of the working paper are delineated. 
Next, nine statewide priorities are presented, relating to the 
following concerns: (1) open admissions; (2) opportunities for 
individual enrollment determined by student readiness and willingness 
to learn; (3) instructional excellence and balance; (4) high quality, 
articulated transfer programs; (5) preparation for gainful employment 
through degree programs, short-term programs, and joint programs with 
business; (C) student guidance and support services; (7) remedial 
education; (8) continuing and community education programs; and (9) 
responses to local needs through community services. The next section 
contains guidelines for institutional evaluation with respect to each 
of the priorities. Next, examples are provided describing 
hypothetical evaluations of three statewide priorities by three 
districts, reflecting an approximation of real conditions and a 
possible analysis and problem identification process. A discussion of 
the Boerd of Governors of the California Community Colleges' role in 
evaluation is appended. (LAL) 
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WORKING PAPER NO. NINE 



EVALUATING STATEWIDE PRIORITIES 



PREFACE 

This working paper on Evaluating Statewide Priorities is one of a series of 
papers resulting from a three-year project to improve evaluation and planning 
in community colleges. The project is sponsored jointly by the Chancellor's 
Office of the California Community Colleges and by the Western Association 
Accrediting Commission for Community and Junior Colleges. Project work is 
concentrated in California and Hawaii, the jurisdiction of the Western 
Accrediting Commission. Support for the project is provided by community 
colleges in these states, the two sponsoring agencies, and by the federal Fund 
for Improvement of Postsecondary Education (FIPSE). 

Project objectives include developing a clear statement of the responsibilities 
for evaluation and planning th;* are appropriate for state control agencies, 
accrediting commissions, and fr • local community colleges. Tensions about the 
appropriate division of these responsibilities exist throughout the country. A 
long tradition of cooperation in California and Hawaii, however, has created a 
most congenial atmosphere in which to analyze and clarify the proper 
delineation of roles. 

• 

Project staff also are developing a series of tools to improve the state-cf-the 
art of evaluation and planning for community colleges. Beginning in the Fall 
1982, these tools have been introduced, used and -assessed in a dozen workshops, 
self-study seminars, symposia, and problem-solving sessions conducted in 
California and Hawaii. These activities will continue through the Fall of 
1984. While project work is being concentrated in the two states, it should be 
possiole to generalize the results to virtually any community college operation 
or governance structure in the country. 

This working paper was written earlier in the project as a guide to 20 
community colleges in California who were undergoing accreditation self-studies 
during 1982-83. These colleges were asked, on a volunteer basis, to evaluate 
their performance with respect to the statewide priorities described in this 
paper. The priorities were 1 derived by project staff from a statement developed 
by the California State Board of Governors in the Fall of 1983. 

These evaluations >are being reviewed by visiting teams as a separate, 
experimental part of the accreditation visit this year. Results of this review 
will be combined with otner sources of data (without identifying individual 
colleges) to produce an aggregate accountability report for state-level 
purposes. 

This part of the FIPSE Project is designed to test, in a practical, cost- 
effective way, the Project's proposed delineation of role where accreditation 
assesses the quality of the community college - the range, depth, and 
effectiveness of its programs and services, its governance, and how well it 



serves students. The state agency, by contrast in this delineation, is 
concerned with statewide objectives and the degree to which colleges, in the 
aggregate, are meeting those objectives. A more general discussion of the 
state agency's possible role in evaluation (again using California) is 
contained in the Appendix to this paper. 

A report on this work will be completed in June 1984. This report will include 

•tuStantlve results of the analysis together with an assessment of the 

? v -iii^nt* as to whether this "bottom-up" approach to aggregate evaluation is 



will note that we, the project staff, have other responsibilities. 
— nseq • Jy, were it not for the help and assistance of countless others in 
both Hc.;aii andCaiifornia, this effort would be impossible. Unfortunately, 
space does not permit us to list all these individuals. However, we do want to 
thank Evelyn Stacey of the state Chancellor's Office and Rich Montori of 
Monterey Peninsula College for their excellent work, respectively, in typing 
the manuscript and in prepariny the art and printing for this document. 

We especially appreciate tue support from FIPSE. Receipt of the Fund's grant 
has set in motion a series of commitments on the part of others whose support 
(in money and in kind) is essential to the succes ful completion of this 
project and the implementation of its results. * 



Chuck Mclntyre 
Project Director 

Director, 

Analytical Studies Unit 
State Chancellor's, Office 
California Communify 
Colleges 



Robert Swenson 
Project Co-Director 

Executive Director 
Western Accrediting 
Commission for 
Community and 
Junior Colleges 



Dale Tillery 

Principal Project Consultant 



Professor Emerit is, 
School of Education 
University of California, 
Berkeley 



WORKING PAPER NO, NINE 
SEPTEMBER 30, 1982 
REVISED JANUARY /, 1983 



PURPOSE 



This wu'Ung paper is intended for use by ti.ose colleges undergoing accredita- 
tion self -studies during 1982-83. These colleges and their districts are being 
asked, for the first time, to evaluate their efforts with respect to at l«ast 
three of the statewide priorities as well as to work on the ten accrediting 
standards, the traditional requirements. 

The paper presents the priorities, some guidelines for evaluation and several 
example analyses. Many different techniques of analysis are possible, depending 
upon the availability of staff and relevant, information. Much of the evaluation 
of accrediting standards covers college practices that are pertinent to the 
priorities. This information can be incorporated by reference in the evaluation 
of statewide priorities, rather than being developed a second tine. 

o 

It is hoped that this paper will stimulate thinking about how colleges may evaluate 
their status with regard to statewide priorities. Results of analyses undertaken 
by self -study colleges during this first, test, year will help identify the most 
effective evaluation techniques and provide the basis for modification of this 
paper. In the meantime, your comments and suggestions on this paper are welcomed. 
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A. VRIORITIES 



1. California community colleges provide open admission opportunities 
to all adult California residents, without regard to race, ethnic 
or national origin, sex, age, disability, or prior educational 
status. 

2. Opportunities for individual enrollments in specific courses and 
programs are to be determined by evaluation of student readiness 
and willingness to learn. ° 

3t Community colleges should achieve Instructional excellence and 
program balance. 

4. Conounity college disuicts will provide high quality transfer 
programs which are carefully „and continuously articulated with 
the senior Institutions and the high schools. 

!>. Community colleges prepare individuals for gainful employment 
through the following programs: 

(a) Two-year vocational/technical programs leading to 
AA/AS degrees which include essential skills and 
general education; 

(b) short-term certificate programs leading to early 
employment and for continuing/reentry education; 
and 

(c) joint programs with business, industry, labor, and 
government (such as Joint Apprenticeship, CWETA, 
Investment in People, cooperative work experience, 
and contract instruction). 
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Guidance and support for students Is an essential function of 
community colleges, including 

(a) assistance in matters of financial aid, placement, 
and program selection; 

(b) academic and career counseling; 

(c) monitoring students on probation; 

(d) articulation with high schools and senior institutions 
and 

(e) coordination with the college's instructional programs 

Relevant programs of remediation should be required for all 
students who need preparation for successful learning in 
community college-level courses and programs. 

Continuing and community education programs should effectively 
respond to such state and local needs as 

(a) adult basic education; 

(b) English as a second language; 

(c) general education; and 

(d) education for citizenship. 

Community colleges should respond to unique local needs by 
offering the appropriate community services. 



B. EVALUATION GUIDELINES 



1. dlf.nl. community college, provide open admission opportunities 
to .11 adult dlfornian, without regard to race, ethnic or 
national origin, .ex, .ge, dls.blllty, or prior educational 



status. 



Ev.l».tlon of this priority begin, by co-paring college enrolment to 
the adult population of the college's ,ervlce area, adjusting for the 
existence of other educational supplier,: <a> over tUe < "0 and 
„80 ceneue, for example). (») 1- »* " l " eS " 

dlfomla. and (c) in relation to norm, determined locally by review 

of coaminity educational needs. 

„„d.rr.pr..ent«d group, would be Identified b, »••»■• « 
,„roll..nt/popul.tlo» r.tlo. for ap.clflc group., defined y location 
.nd by .ocl.cultur.1, economic, and de.ogr.phlc (race, ethnicity, ge. 
g ,„d«r) .t.t».. R«.on. for und.rr.pre.,n=.tion would be Identified 
and ana ly ted. 

(See example analysis.) 



2. opportunity, for individual enrollments In .peclflc course, 
and program are to be determined by evaluation of student 
readlne.a .nd willingness to learn. 

Specific strstegle. ..ployed by the college, to evaluate, counsel 
.4.1., and direct student. Into cour.e, and program, . ould be 
.n.lysed. Thl. includes way. college, are 8 
opportunities for student, with special need, ,uch a, the dls.bl d 
,.c.d..lcaUy unprepared or less capable, and economically 
disadvantaged. 

10 



If the college/district needs to ration the supply of spaces in 
specific course or program enrollments! is this done by eliminating or 
closing courses, closing enrollments after a certain level, or by 
evaluation and screening of students prior to enrollment? 

Much of the analysis of this priority is undertaken under Accrediting 
Standard 2C.3, dealing with student learning capabilities, and under 
Standard 4 which covers a variety of student services, including 
counseling, 

3. Community colleges should achieve instructional excellence and 
program balance. 



Instructional excellence can be evaluated largely through the 
standards of accreditation and by reference to measures of learner 
outcomes. The specific assessment of instructional excellence can be 

c 

accomplished by use of Accreditation Standard! 2, 3, 6, and 7, dealing 
with programs, staffing, learning resources, and facilities. 

The notion of program balance refers to the need to match college 
curricula and services with the needs of individuals in a college's 
service area (see Accreditation Standard IB) and with the present and 
future structure of local and regional labor markets* Evaluation of 
the college's progress toward statewide priorities must Include review 
of local effort? to meet community needs with due recognition of other 
local educational suppliers. An effort should be made to assess 
expected future, as well as current, needs and preferences. * c t 

Any one or all of several classifications may be used to group data 
whery instructional programs are being compared with the needs and 
preferences of those in the college/district's service area: 
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(a) by student type: 
disabled 
disadvantaged 
other; 

(b) by student objective: 
transfer 
occupational 

^ first employment 
upgrading 

retraining and/or career change 
othetj. or 

(c) by subject area of program: 
aggregated from TOP* CIP or other 

" taxonomy. 

(See example analysis.) 

i 

r 

' ,4. Community colleges will provide high quality transfer programs 

which are carefully and continuously articulated with the senior 
institutions and the high schools. 

j - 

•The quality of this program can be determined through accreditation, 
•by reviewing the competence of staff, adequacy of the curriculum, 
"level of-counseling, and the effort by the college at articulation 
with high schools and senior institutions. 

Specifically, transfer programs are evaluated under Standard 2A.4, 
articulation efforts under Standard 4A.3. and counseling under 
Standard 4A.6» 

v 

Quality may also be measured by the progress (i.e., intra-term, 
' Inter-tew, lnter-tfear persistence) and performance (g.p.a., cognitive 
and affective, skills added, etc.),of students enrolled in transfer 
. programs. 

- v ^ : . 12 



Outcomes can be measured by the rate and success on the part of those 
Who desire c to transfer both (a) over time In the college and (b) In 
comparison with other colleges. The rate compares thosu fro* a cohort 
of students who actually transfer with those from the same cohort who 
declare they wanted -to transfer or who declared and Rook the needed 
courses (excluding -those who took the wrong course pattern). Success 
may be measured by comparing the g.p.a.'s, time-to-degree (in upper 
division) and subsequent citizenship, employment or advanced education 
of transfer versus "native" students, adjusting for students' atllity 
(original eligibility, for example). 

(See example analysis.) 

5. Community colleges prepare individuals for gainful employment 
through the following programs: 

(a) Two-year vocational/technical programs leading to 
AA/AS degrees which Include essential skills and 
general education; 

(b) short-term certificate programs leading to early 
employment and^ for continuing/reentry education; and 

(c) joint programs with business, industry, labor, and 
government (such as Joint Apprenticeship, CWETA, 
Investment in People, cooperative work experience, 
and contract instruction). 

Institutional and student performance can be eyaluated in the same way 
for each ot the three areas, provided that each subgroup of students 
is accurately identified. Such identification would be based both on 
student declaration, demographics, and pattern of courses taken. Once 
identified, students may be assessed, during enrollment, by measuring 
the change (from initial enrollment to completion in 
cognitive/effective attributes or In essential/general skills. This 
Implies the use of standardized competency tests at entry and exit. 
Proxies for measuring student progress Include persistence (intra- and 
inter-term) and performance (g.p.a. f etc*). 

13 
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Another assessment of program quality U accomplished by use c? 
Accreditation Standard J*oa. 2 (particularly 2A.5), 3, 4, 6, and 7 
dealing with programs, staffing, support services, learning resource*, 
and facilltiea. For instance, are learning resources efficiently 
utilised and up-to-date with current technologies? 

A comple :e evaluation also requires follow-up study of the effects of 
these programs after students have completed their work. Among the 
relevant measures are the number and/or rate of job placements, 
advancements,' or improved productivity among those who complete these 
programs. Do students work In the field for which they are trained? 
Also of particular Interest is the relative ability of these 
individuals to adapt to changes in technology or In the labor market. 
Finally, are the general skills imparted such that students can 
function as responsible citizens? 



6. Guidance and support for students is an essential function of 
community colleges, including 

(a) assistance In matters of financial aid, placement, 
and program selection; 

(b) academic and career counseling; 

(c) monitoring students on probation; 

(d) articulation with high schools and senior 
institutions; and 

(e) coordination with the collet's Instructional programs. 

This evaluation is conducted largely under Accreditation Standard 4, 
particularly those components dealing with articulation, 4A.3, 
counseling, 4A.6, and financial aid, W.14. Information from the 
self-study can be consolidated and reworked for this evaluation. 
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7. Relevant programs of remediation should be required for all 

students who need preparation for successful learning In community 
college-level courses and programs. 

Several questions need to be answered In this evaluation. Do such 
programs exist at the college? If so, are courset well-designed and 
well-taught? Are guidance, counseling and probation used to help 
students? Are programs articulated with other institutions? 

Do programs serve those with deficiencies in learning skills and 
special learning problems, as well as those students who simply lack 
adequate preparation for specific courses? 

Also of importance is evidence that the college has examined 
alternative Instructional methods such as self-paced instruction and 
mastery learning. 

Most of the analysis here is performed during accreditation, largely 
under Standards 2A.6 and 4A. 14-18. 

8. Continuing and community education programs should effectively 
respond to such state and local needs as 

(a) adult basic education, 

(b) English as a second language, 

(c) general education, and 

(d) education for citizenship. 

Evaluation measures for this priority are analogous to those for 
programs preparing students for employment. A major difference, 
however, Is the greater need here to accurately assess (a) the 
specific needs of the community (those within the college's service 
area) and (b) the exlste* •;. of other local suppliers (such as K-12) of 
these same educational services- Another difference Is the use of 
measures of citizenship, rather than employment, to assess learner 
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outcome*. Such measures include political participation, charitable 
and civic work, infernal education of children, consumer capability, 
and general communication skills, among others. 

information from the self-study of Accreditation Standards 2A,5 and 5B 
la particularly relevant here. 

\ 

9. Co«unity colleges should respond to unique local needs by 
^ offering the appropriate community services. 

Evaluation of this priority can be accomplished by summarizing the 
.elf-atudy Information developed under Accreditation Standards 5A 
through 5F. 



a 
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y C. EXAMPLE EVALUATIONS 



The following examples describe hypothetical evaluation* of three 
statewide priorities by three districts. While hypothetical, the 
examples were constructed to approximate real world condition!. Each 
e-anple presents a possible analysis 'and problem identification 
without attempting to present solutions. 

Please review the examples with the following question* in mind. 

1. Are data available for « similar analysis in your district? 

2. If similar data are not "available, what kind of proxy 
information might be used? 

3. If data are available, how would an analysis of your 
district differ from the examples? 

4. What important factors have, been omitted from the 
evaluation? 

5. What statistical or other analyse* may be used for 
deriving conclusions of the Kind shown? 
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Priority No. I. California community colleges provide open 

enrollment opportunities to all adult Calif ornians, 

without regard to race, ethnic or national origin, 
sex, age, disability, or prior educational status. 

General 

District X is an urban multi-college district with responsibility for 
continuing adult education in its community. Its 
enrollment/population (E/P) ratio in 197 0 was greater than both the 
statewide average (expected in view of "the adult education 
responsibility) and the average of six similarly situated districts 
(not necessarily expected) as shown in Figure A. 

While the '70 enroll sent coverage was relatively high, the trend 
since, particularly prior to Proposition 13 (1978), has lagged. Had 
the district increased its enrollment in a fashion similar to its 
comparison group, it would have enrolled one of every five adults or 
20Z of its population. This is some 15,000 students greater than the 
actual enrollment during Fall 1981. While the X's .12 E/P is 
significantly higher than the statewide E/P of .08, district 
policymakers conclude there has been a deterioration in their 
enrollment performance, Why has this occurred? 

Several factors appear to account for much of the E/P drop: 

(a) A stable and aging service area population; P grew by 
SZ between 1970 and 1980 as opposed to 151 in the 
comparison districts and 25% statewide, 

(b) the emergence of numerous proprietary institutions 

in district X's service area, particularly after 1975, 
and 

(c) development, throughout the 197 0's, of programs in local 
adult schools. 

18 
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Factoring out the Impact of (a), (b), and (c), the district finds that 
there lis still s significant drop In Its E/P ratio relative to 
comparable districts as shown by area (d) in Figure B. 

Further analysis reveals that this gap (d) can be explained largely by 
three additional factors: 

(1) a decline in veterans enrollment following 1975 that was 
not experienced to the same degree in either the comparison 
districts or all other districts, 

(2) a significant shift from family to single or child-less 
households between 1970 and 1980, and 

(3) a significant influx of Indochinese refugees beginning 
in 1978. 

Specific 

A parallel analyaia is made by district X of varioua subpopulations in 
Its eervlce area with the following conclusions: 

(a) after large increases in the early 197 0's, the dl trict's 
E/P ratio among those below the poverty income level 
declined significantly after 1974, 1 

(b) the E/P ratio among the district's disabled population 
Increased dramatically since 1974, 

(c) Asians and Hlspanics are enrolled at a rate far below 
that of both "comparison" and "other" districts, and 

(d) enrollment rates of blacks, caucasiana, and females are 
<' comparable to those In other districts. 

Needs Assessment 

Consistent with Accreditation Standard 1B.1, district X conducts an 
educational needs assessment of those in its service area. The 
resulting distribution of instructional programs and support services 
preferred ("needed") is then contrasted with the distribution of 
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program and services offered by district X, adjusting apparent 
discrepancies by the existance of other local educational suppliers. 

In summary, district X offers relatively less ESL, remediation and 
courses for the elderly than are preferred by Us clientele. Unmet 
financial need appears large. Upon further analysis, it is shown that 
district X provides relstively fewer EOPS services to its EOPS 
eligibles and maintains a i.ar smaller financial ald.staff than is the 
case in the six "comparable" districts or in another set of ten 
districts with comparable proportions of their clientele below 
poverty. Certain of these programmatic "gaps" appear related to 
factors noted In the enrollment analysis. 1 

(A aore specific and detailed comparuoi* of district X's programs and 
services with the programs and services needed by its clientele is 
performed for the statewide priority that deals with "program 
balance." In that case, other dsta on local labor market structure 
are also analys . > 



Projections 

Relying on extrapolation of recent demographic trends, local planning 
agency projections, and ressonable assumptions about future local 
econom' development, growth in nearby suburbs, and refugee 
laal? t ,cn, district X concludes that 

) j;otal district population will decline and age until 

1988 after which these trends will be reversed, 
}) the trend toward single and child-less households will 

continue throughout the 1980' s, 
O future economic growth will take piece in certain highly 
technical Industries and will, by 1990, require skills 
not now in existence, ai*d 
(d) refugees will coaprise an ever- Increasing proportion of 
district population. 

21 



It It likely that, absent eigniflcsnt policy changes, the relative 
(and absolute) decline In district X's E/P ratio will continue for 
another five years then level soaevhst. 
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Priority No. 3. Community colleges should achieve instructional 

excellence and program balance 



District^ Is a relatively small, older college In the inner city. 
District population la stable and unemployment ha- increaaed 
significantly during the past three years. 

/ 

/Instructional Excellence 

An assessment of program quality is accomplished through Accreditation 
•standards 2, 3, 6 and 7, dealing with educational programs, staffing, 
learning resources, and facilities. Questions are raised concerning 
relatively large class sizes in many of Z's occupational labs and 
shops, small class sizes_in second-year courses leading to transfer, 
and a relatively large proportion of equipment that is obsolete and 
not well maintained. Many of these observations are derived following 
initial comparisons of Z's programs with those of six similar 
sraall-to-moderately sired Inner city colleges. 



Program Balance 

Fiirthir analysis is made of the results of an extensive survey of 
professed community educational nee^s along with a review of the local 
and\ regional labor market structure. A distribution of preferred 
programs Is then developed and contrasted with a distribution of the 
Instructional programs offered by Z (see Figure C). \ 

Relative /to/ the apparent need , 'd le^rlct Z has/allocated -too few 
resour^es((4Sjjieasured by faculty course, assignments) In six of eight 
occupant l^ajl /areas, the natural sciences and/adult basic skills. At 
the sam^t^e 7 . It appears that too many di s/trlct resources may have 
been allocated to social science, humanities and fine arts, and 
distributive (occupational) education. 

Further analysis reveals the existence in the district of a number of 
proprietary institutions offering programs in the occupational 
programs of Justice and business. Consequently, Z appears deficient 
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FIGURE. C 
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DISTRICT Z COMMUNITY PREFERENCES 



DISTRICT Z FACULTY COURSE LOAD 



CLASS 

SIZE: 35 20 15 26 
Stw: 27 30 25 24 



25 23 - 35 30 30 25 22 
25 23 20 27 25 20 22 26 



25 20 20 18 
20 20 20 18 
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f 

only in health, technology, trade and Industry, and consumer/home 

economics. 

c- 

It U also noted that in those programs where Z appears to have 
allocated relatively few resources, Z's class sizes exceed those 
experienced in community colleges elsewhere. Likewise, for 
"over-allocated" programs, Z's class sizes are less than those of 
similar programs elsewhere. 

Further review is undertaken by district Z of accreditation findings 
bearing upon those programs that appear most "out-of-balance." 
Program strengths and weaknesses uncovered by the use of accreditation 
standards tend to parallel the analysis of balance. That is, 
relatively more weaknesses are uncovered In those programs in which 
the faculty allocation is relatively low and in which class sizes are 
relatively high. 



25 



. - 21 - 




Priority No. 4. Community college districts will provide transfer 



District Y is a middle-sized (14,000 enrollment) district in an older 
\ suburban setting that has already experienced its major population 



Reviiw reveals that, like other areas of the state, the district's 
lumber of annual transfers to UC and GSU have declined since 1976. 
This decline is modified to a degree by an increasing proportion of 
Y's transfer students going to private and out-of-state senior 
institutions. (Data he^ are limited to just three years, however.) 
Like other areas of the state. Y's 18 to 24 year-old age cohort has 
decreased since 1978 (see Figure D). 

Using the Statewide Longitudinal Study, annual district surveys of 
student objectives, numbers of full-time students, and other 
indicators/evidence, the district estimates that the number of 
students enrolled In the transfer program increased to 1975 and has 
since decreased to a level slightly below that of 1970. 

Comparing the estimated numbers of actual transfers to those enrolled, 
the district finds that the rate of transfer from among those enrolled 
for this purpose decreased between 1970 and 1975 and has since 
Increased to a leve\ greater than that of any year during the 1970' s. 

Similar data from a set of ten comparable suburban districts are not 
available, but a casual review suggests that their experience since 
1975 has been lower than that of District Y. 

Further study of a limited amount of longitudinal data, however, 
reveals significant instability In the group enrolled for transfer. 



growth. 



Rate 
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FIGURE D 



District population 




18-24 year-old cohort 
in district Y 



NUMBER 




^>-^ Estimate of .potential 
transfer students 



>A Estimate of total transfers 



UC, CSU transfers 
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That la, fawer than 30Z of the transfer group persists beyond their 
first year, a level that Is significantly below estimates of the like 
rate statewide and In the ten comparable districts. Further, those 
who do transfer spent an average of three years In lower division at 
Y. This exceeds similar t ima-to-completion rates elsewhere by 
one-half year. 

The results appear ambiguous. The relatively low Inrer-term (and 
poaslbly intra-tera) rates of students progress august that the 
tranefer/enrollment rate may overstate dlatrlct Y's performance. 
Further research Is needed, but, In any case, the data on student 
progress are Important to program evaluation, the topic of Accrediting 
Standard Ho. 2 A. 4. 

Success % 

District Y has close contact with tt< , UC and CSU campuses to which 
Y's atudents typically transfer. Uttlm Is known about other 
four-year schools to 4ttch Y's students transfer. 

Attempts to secure data on Y's transfer students at UC and CSU are 
relatively successful. Efforts to obtain comparable data on transfers 
from the ten comparable diatrlcts are less successful. . 

Analysis shows that, on the average, Y's students experience a g.p.a. 
drop of 0.5 point during their first term after transfer to UC. By 
graduation, however, Y's transfers have improved their g.p.a. by about 
one point to a level that exceeds the average g.p.a. of native 
students. The average time-to-completlon and the rate of completion 
of bachelor's degrees on the part of Y'a tranafers, are significantly 
below that of native students, however. 

* 

Review of Y's transfers to CSU reveals even higher (than UC) 
comparative results on g.p.a. (compared to natives), but similar 
.results with respect to time-to-completlon of bachelors' degrees; that 
is,' Y^s transfers at CSU take significantly longer to complete upper 
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division than do CSU native students. 



Efforts were partially successful In separating and analysing those 
who were originally eligible (for CSU or UC out of high school) in 
contrast to those who were not eligible. Notably, Y's transfers 
appear to be s>ade up of a much greater proportion of 
"originally-eligibles" than Is suggested for community colleges 
generally from the&few studies available on this topic. The 
higher-than-average g p.a. rates of Y's transfers in upper division 
appear to be partly t >lated to this phenomenon. 

Programs 

District Y's efforts at articulating transfer programs with ot^r 
Institutions are assessed in Accreditation Standard 4A.3. 0-her 
support services, such as counseling and guidance, are reviewed in 
other components of Standard 4 and in relation to the statewide 
priority dealing with student support services. 

These analyses reveal a number of areas where improvements could be 
aade in both the articulation and counseling functions,. In several 
cases these findings appear to relate to the findings on transfer 
student success (noted above). 
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Item 

Summary 

This is an information item designed to help the Board begin discussion of its 
role in evaluation. Current efforts to evaluate the performance of California 
community colleges range from state-level review by the Chancellor's Office on 
behalf of the Board -o work by the colleges on internal evaluation of 
programs, activities and staff. Of major importance is the evaluation of 
colleges thU: is conducted through accreditation. 

Recognition of the need to explicitly define the roles of accreditation and the 
Board of Governors in evaluating]ttte;work of the community colleges led to the 
project supported by the federal Fund tor Improvement of Postsecondary 
Education (FIPSE). Vlork of the FIPSE project should help the Board determine 
how its evaluation activities are distinct from accreditation and how the 
Board's activities can support the Board's responsibilities for leadership and 
advocacy for community colleges. 



Background 

The Board of Governors is responsible for leadership, advocacy, and 
regulation of California community colleges. These responsiblities involve a 
continuous series of decisions about public policy. While districts and 
colleges are responsible for the essential functions of teaching and community 
services, the Board works to protect the rights of citizens and to oversee the 
use of public funds. 



Board decisions typically involve identifying statewide educational needs, 
assessing how well these needs can and are being met by community colleges, and 
what pol icies can contribute to improvements in the way col leges meet theso 
needs. This assessment invariably raises questions about the Board's role in 
evaluation. While community colleges have a long tradition of local autonomy 
and responsibility, there have been increasing pressures for greater 
accountability most often focused at the state level and on the Board, 
particularly since the passage of Proposition 13. 

The Board's continuing interest in improving accountability was highlighted in 
its 1979 resolve to undertake a series of steps that would result in more 
effective planning ana compliance. More effective planning is being sought 
through wc k by the Board and staff on Comprehensive Planning (see item 
elsewhere inj* this agenda). Compliance is being improved through actions which 
have simplified provisions of the Education Code and Title 5 of the 
Administrative Code and, most recently, in a comprehensive review by the Board 
of minimum standards entitling districts to receive state aid. During one of 
these Board deliberations on minimum standards, a number of concerns were 
raised about 1 accreditation. 

The importance of accreditation and the need to coordinate this activity and 
the work of the Board of Governors gave rise to the FIPSE project. The project 
is designed to generally improve community college evaluation and planning. A 
major specific objective of the project is to delineate jthe respective roles 
that the Board, the Accreditation Commission for Community and Junior 
Colleges,, and the districts and colleges are to play in evaluation. As work 
proceeds, care is being taken to distinguish clearly between evaluation and 
compliance. 

Compliance has to do with the community colleges meeting certain minimum 
conditions for the conduct of their operations and in their use of public 
funds. If these minimum conditions are not met, sanctions are introduced to 
ensure that compliance is forthcoming. Evaluation, by contrast, deals with the 
quality of community college efforts to achieve goals and objectives that are, 
for the most part, substantially higher than the minimum conditions associated 
with compliance. If goals and objectives are not being met, new or changes to 
existing policies are recommended which will aid and encourage their 
achievement. Evaluation includes not only assessment of existing performance 
but also identification of ways to improve that performance. 



Definition 

Evaluation has been described by Paul Dressel as "...both a judgment on the 
worth or impact of a program, procedure, or individual, and the process whereby 
that judgment is made." The term evaluation can be used to describe the 
activity, its end product, or both. 
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The activity of evaluation can be accomplished in a variety of ways, ranging 
from an extensive, very rigorous analysis to a very cursory, subjective 
consideration. These respective approaches are sometimes distinguished by the 
terms "hard" evaluation versus "soft" evaluation. In hard evaluation, analysts 
attempt to identify causal relationships between inputs and outputs. This kind 
of work typically involves subjecting objective data to sophisticated 

statistical analysis. By contrast, efforts in soft evaluation are aimed at 

identifying correlational, rather than causal, relationships, and are based 
very often on subjective data without resort to heavy statistical analyses. 

Evaluation also may be categorized as "formative" or "summative". In formative 
evaluation, the analyst often works closely with the program manager or 
policymaker while the program or policy is being implemented. In this way, 
results of the evaluation can be translated quickly into improvements in the 
effort prior to its completion. By contrast, summative evaluation focuses on 
the end product or outcomes of an activity and may take place after the 
activity is completed. In this case, the evaluator is an objective observer of 
the activity, rather than beiny a participant in its conduct. In practice, 
however, these distinctions are blurred, since even most summative evaluations 
are undertaken for the explicit purpose of improving policies or programs at 
some point prior to their completion. 

For most evaluations, it is accepted practice to identify and attribute values 
to the goals and/or objectives of the activity under scrutiny. The analysis 
then proceeds so as to determine the extent to which the activity is achieving 
these values. An alternative (though a less frequently used) technique is "goal- 
free evaluation," which looks at actual outcomes of the activity without regard 
to the intended outcomes. Here, unintended or side effects of the activity 
become an implicit part of the analysis. In practice, evaluations which are 
goal- or objective-based, should incorporate side effects if they are to fully 
measure the consequences of the activity being examined. 

In normal use, the term -goal refers to values that are typically global 
descriptions of ends that are not constrained by time. By contrast, objectives 
are specific, usually measureable, to be achieved by a certain time, and 
^descriptive either of means or ends or both. In practice, analysts settle for 
values that fall somewhere between goals and objectives. Use of the term 
"priorities" in the Board statement developed this past fall seems to be a 
compromise of this sort. 



Existi ng Evaluation 

Responsibility for the evaluation of student performance, the effectiveness of 
courses, programs and services, and the work of staff resides at the district 
and college level. Evaluation of the institution or college is the 
responsibility of the regional accreditating agency. For California community 
colleges, this agency is the Accrediting Commission for Community and Junior 
colleges, a part of the Western Association of Schools and Colleges (WASC). 
Certain programs also are examined through specialized accreditation by 
professional accrediting bodies and by the Board and Chancellor's Office in the 
case of those programs that are categorically funded by the stute. The Board 
and other state bodies conduct sporadic analyses of how well the colleges are 
performing as a group, 
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As always, evaluation of student learning iis the responsibility of faculty 
assigned to the classroom. There are, however, efforts underway that are aimed 
at improving the assessment of students that takes place outside the 
classroom. Among such efforts are the highly publicized "Miami Dade model M and 
the implementation, in New Jersey, of state-mandated assessment and placement 
testing in community colleges. In California, work by the Learning Assessment 
and Retention Consortium (LARC) is directed toward preparing models that can be 
used by colleges to assess the readiness of students for particular courses. In 
addition, LARC is working on alternative ways for colleges to conduct and 
evaluate remedial programs and courses. Another aspect of the work by LARC s is 
a followup mechanism to evaluate changes in the retention and performance of 
students. Work by the Chancellor's Task Force on Academic Quality may also 
consider student assessment techniques. 

Most districts and colleges conduct program evaluations which result in the 
modification or elimination of existing courses. These evaluations typically 
involve an extensive analsysis of program costs and the revenue forthcoming to 
individual programs through the apportionment process. Less often, these 
evaluations are extended to look at enrollment trends that are anticipated in 
individual programs. Only rarely do these evaluations include efforts to assess 
the quality of programs. 

While accreditation is focused on the instituition, not on individual programs, 
the self-study does determine if program review is taking place and how well it 
is carried out. The focus of accreditation is best described tfy its two 
fundamental purposes: 

(a) to assure the quality of the institution and 

(b) to assist in the improvement of the institution. 

All regional accrediting bodies, including the Western Commission, use a common 
process. This process involves first a self-study of the institution, using 
Commission standards, by staff at that institution. This self-study is followed 
by the on-site visit by a team of peers in order to validate findings of the 
self-study and to make recommendations for improvement. The team's report, 
including recommendations as to the college's accreditation, is reviewed by the 
Commission and a final decision rendered. 

The college is accredited by the Commission which certifies that the 
institution (a) has appropriate purposes, (b) has the necessary resources, (c) 
is accomplishing its purposes, and (d) will continue to do so. 

The Accrediting Commission encourages colleges to make public both the self - 
studies and the evaluation of team reports after the Commission has taken 
action. However, disclosure of the content of the reports is a decision that, 
by policy, belongs to the institution being accredited. . ,, 

Accreditation is a prerequisite for the college to receive public funds and for 
its students to receive federal financial aid. Because of the quasi-public 
nature of accreditation, public members sit on the Commission. Commission 
policy meetings are open to the public. Actions on both standards and on 
accreditation for individual colleges are made public. At the same time, the 
voluntary, self-regulatory nature of the process is the basis for a measure of 
confidentiality. Institutions make full disclosure to the Commission which, in 
turn, keeps the information confidential. Some of this information is quite 
sensitive, particularly if it involves college personnel. In any case, full 
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disclosure of both self-studies and team reports would reduce their candor. 
This candor is a. ma jor basis for efforts at improving the institution. 

Increasingly, there is an effort to focus the accreditation self-study on 
outcomes, rather than on inputs or process as has been «the case traditionally. 
This emphasis can be seen in the new standards being used by the Commission- for 
community colleges in California, Hawaii, and the Pacific Basin. Under these 
standards, colleges conducting self -studies are encouraged to examine 
critically their performance in the areas of college goals, educational 
programs, staff, student services, community services, learning resources, 
physical facilities, financial resources, governance and administration, and 
district and college relationships. Experience to date with the new standards 
suggests that their, use results in a more structured and more rigorous self- 
study. 

The Commission works on a five-year cycle. Consequently, a community college is 
subject to review at least once every five years if its performance is 
satisfactory and more often if there are difficulties or changes requiring 
• followup visits. Costs for the Commission and the expenses of visiting teams 
are estimated at nearly $200,000 each year for about 25 to 30 college 
accreditations. The cost cf college staff time involved in the visits and,' 
more importantly, in the self-study add to this cost. 

Staff of the Chancellor's Office participate on each of the accrediting site 
teams and on the Commission itself. In addition to that, the Office performs 
evaluations of the three major categorically funded programs: 

(a) . Extended Opportunity Programs and Services (E0PS), 

(b) Handicapped Students Programs and Services (HSPS), and 

(c) Vocational Education Act activities (VEA) 

as well as the evaluation of projects submitted under the provisions of the 
State Fund for Improvement of Instruction. 

o 

For E0PS, data on student enrollment and activity and program expen ires are 
analyzed to determine if college programs are meeting the needs of -indents and 
if the programs are efficiently managed. In addition, operational program 
reviews are scheduled for two to four-day periods throughout the academic year 
to provide formative information to E0PS administrators and to assist in the 
systemwide analysis of EOPS. Review teams address three questions: 
"(a) What has the program accomplished? 

(b) What are the characteristics of EOPS students? 

(c) How do program activities compare with the program plan? 

These reviews are for the purpose of program improvement rather than <;ing 
vehicles for determining compliance. A similar format is being developed for 
the evaluation of handicapped programs. 

VEA activities are evaluated by Chancellor's Office staff largely through two 
vehicles: the Community College Occupational Program Evaluation System (COPES) 
and the collection of information on students after they complete or leave a 
program, generally termed "followup," which is required by the vocational 
education data system (VEDS). 

COPES relies on the use of some 40 indicators of program quality. Using these 
indicators, college staff and students are asked to record their perception of 
how well the college is doing. Responses are processed and provided to a 
visiting team of peers who validate the" results (somewhat like accreditation). 
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Beyond these activities, Chancellor's Office staff perform a number of other 
activities, such as reviewing facilities utilization, which could be considered 
as evaluation. Aside from various ad hoc analyses, such as the various studies 
involving transfer students, however, theri is no framework for consistent and 
comprehensive evaluation of the performance of the colleges as a qroup. For 
instance, there is no Chancellor's Office evaluation of the, vast arry of 
nonoccupational programs that make up over half of the course activity in 
community colleges. Nor is there any consistent evaluation of student support 
services by the Chancellor's Office. 

Each year, the Chancel lor-' s Office reviews annual information submitted in 
contracted district audit reports. These reports deal with the use of public 
funds and are matters of compliance. There is no asse^ment of the current or 
projected future fiscal status of districts in a fashion that might parallel 
the evaluation of district fiscal capability that takes place during 
accreditation. Some fiscal evaluation, however, does take place under the 
rubric of school finance legislation or such efforts as the Board's Long Term 
F.inance Plan. 

In much the same ad hoc fashion, other agencies of the state, such as the 
Postsecondary Education Commission (CPEC) engage tn evaluations of broad 
aspects of community college operations. On.-going work on the status of the 
transfer function and the recent work on remedial programs are just two 
examples of CPEC evaluation. 

Aside from VEA support, most federal investment in community college education 
comes by way of the financial aid appropriated for students. Consequently, 
federal concerns about evaluation have to do largely with' the reliability of 
accreditation as a vehicle for determining the eligibility of an institution 
and its students to receive aid. 



Current and Proposed Activities 

The high cost of effective evaluatton, gaps in accountability by community 
colleges as a group for use of public funds, the desire for a strengthened 
accreditation process, and a good deal of apparent duplication in the 
information required by accreditation and by the Chancellor's Office, among 
other factors, resulted in the FIPSE project. 

The FIPSE project proposal called for an explicit delineation of the roles to 
*be played by the state agency (Board and Chancellor's Office) and Accrediting 
Commission in the evaluation of community colleges. Some of the general 
differences between these two bodies are: 

(a) The focus of regional accrediting commissions is on the in- 
stitution, while the focus of the state agency is on state- 
wide objectives and accountability for the expenditure of 
public funds. 

(b) Accrediting commissions are private, voluntary, non-govern- 
mental associations of institutions which formulate and 
subscribe to certain standards. State agencies are responsible for 
implementing the substance and intent of the public laws 

under which institutions may operate. 

(c) Regional accrediting commissions are concerned, in reviewing a 
new program, with the program's relationship to the institution's 
mission, whether the institution has the resources and pro- 
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cesses to initiate, monitor, and maintain the program, and 
whether the program is of writable quality. States have an 
additional concern - the need for the program in terms of state- 
wide objectives and the availability of existing programs, 
(d) Reyional accrediting commissions have quite similar policies and 
procedures, whereas the 50 state agencies vary widely in their 
methods and deyrees of regulation. 

Drawing from such differences in interest and function between the Board of 
Governors and the Accrediting emission, the TIPSE project suggested a broad 

th! n r 6 ^ 10n 1n Wh j Ch the accreditif HJ agency, in general should a sess 
the nternal efficiency of a community college: the range, depth, and quality 

llrlll ?< ri^In? P H° 9ramS a ? d services . its management, and how well it 

itself wlM J tS a and c ?™J un ^ y - The state a 9 enc *» tnen » should concern 
itself with external or statewide objectives and the extent to which the 

examinld In t^KprofecT ° bjeCtiVeS ' " " ^ Mm ^ iS ^ 

2^5^ if appropriate ' needs t0 be detailed ' — 

^^arr^ 1 ^ 1 ? 8 a draft delineation ^ r future consideration by the Board 

the ^^ r;"; 0 !' SUff W J ]1 COntinue work on the experiment " which 

S!'„ST?« statement °f M . lsslon and Statewide Priorities is being tested for 
A . 1 u evaluatlon * In this experiment, colleges undergoing accredition self- 
studies this year have been asked also to voluntarily review at least three of 

2^ U JS 1de P r1 ? rities in additi ™ to the work normally required In the self- 
Paper^No I)" " ^ ^ dMCr1bed in Pr ° j6Ct 

i!2?!S!i t1 ! n de y el °P ed in th is experiment can be aggregated (without identifyino 
individual colleges) with a focus toward the priorities. Other information can 
also be brought to bear from the Chancellor 's Off ice Informat o S?™em from 

etw tl ! d fL S f a ! d fr0m ° the u r S0urces 50 as t0 P ut to 9 ether a compfehen ive 
evaluat on that focuses on how well the community .colleges as a qrouo a~ 
performing with respect to the statewide priorities. 9 P 

J!n^i eXperiment is successf ul. this approach will be recommended as a 
central, on-going component of the Board's role in evaluation. 
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